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Section 1 



INTRODUCTION 

This report deals with the testing of a **test,** The Hest" 
is called COPES {Coninunity College Occupational Programs Evaluation 
System), Created In 1971 by the Chancellor's Office of the California 
Conmunity Colleges with the support and participation of local c<Mn- 
munity college leaders In the state, COPES* timeless goal is to Improve 
the quality and availability of occupational education at California's 
comniunity colleges. 

Working toward that goal, COPES* first two years were devoted 
primarily to development, refinement and application of evaluation 
instruments and procedures. By the end of this period, 31 colleges had 
participated In the COPES process, which combines college self-appraisal 
with validation of the self-appraisal by a visiting COPES team made up 
of California conmunity college professionals and knowledgeable lay 
persons. 

Each participating college hoped this two-way test would enable 
It to secure accurate Identification of occupational education strengths 
and needs for improvement to assist in future Institutional planning and 
allocation of resources. Through COPES, each college had certain 
strengths and Improvement needs identified. 

Yet a question remained: How accurate were the Identifications? 

The answer would depend on the soundness of COPES, on whether 
Its testing mechanism was as objective, as solidly based on measurable, 
quantifiable elements as had been Intended. 

The answer also would determine whether COPES, In its existing 
form, was really capable of being helpful to the participating colleges, 
really constituted to n»ke progress toward Its goal. 

3 
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For 1973-74, a special COPES study, sponsored by the 
Chancellor's Office, was designed to seek that answer: 

The objectives of the study were to: 

• Test the statistical reliability (accuracy and depend- 
ability) of the COPES instrumentation and nrethodology 
through replicated procedures at five colleges. 

• Develop recommendations for strengthening the existing 
COPES design as Indicated by the results of the testing. 

As proposed, the testing was accomplished by utilizing two 
independent site visit teams at each of the five colleges so that the 
degree of agreement between two groups, separately considering the 
same information and following the same procedures, could be ascertained. 
The teams, consisting of six members each, were planned to be of parallel 
composition, in the standard COPES make-up, including conriunlty college 
generalists {e.g., presidents, deans of Instruction), occupational 
education deans and occupational education teachers, as well as lay 
persons and Chancellor's Office representatives- 

The five cooperating colleges were chosen from among the total 
of 14 which had requested 1973-74 COPES evaluations. Although the 
project proposal originally had called for random selection, it was 
subsequently agreed that this approach would not be necessary, since 
the aim was to test the COPES system rather than the colleges. The 
important factor in selection, it was decided, was institutional size. In 
order that the simultaneous presence of two visiting teams could be 
readily accommodated. Thus, the concentration was on colleges of medium 
to large enrollments. 

At each college, members of each team were paired with their 
counterparts from the other team to share all Inputs of a regular COPES 
site visit schedule (e.g., interviews with staff, students and persons in 
the community, and analysis of pertinent college documents and materials). 
These inputs » however, were the only significant things shared by the 



pairs; they were specifically cautioned against sharing perceptions. 
Indeed, aside from the time spent as pairs, the first-night overall 
orientation session with the site visit chairman and such affairs as a 
college-hosted luncheon, there was little, if any, contact between the 
two teams and no conversation dealing with the substance of their 
assignments* 

Upon completion of the interview and study schedules, each 
team independently reached consensus on the 60 evaluation items contained 
in Form 7 {Summary Profile by Site Visit Team) of the COPES instrumentation. 
Ratings assigned each item were based on COPES criteria statements. 

Subsequently, the ratings of the two teams involved In each site 
visit were compared, and exploration was conducted concerning the causes 
of any significant differences in team perceptions. 

Then, in the light of the findings, recommendations were formulated 
for strengthening the COPES design. 

Hopefully, the result of this undertaking will benefit not only 
COPES but the community colleges of California for which COPES exists— 
and, nnost importantly, the students and* communities for which the 
colleges exist. 

Grateful acknowledgment is made to all who participated in the 
study. The five cooperating colleges are identified in Appendix A. Names 
and affiliations of site visit team members are shown as Appendix B. 

This report is organized to deal first in suimtary form and second, 
where warranted, in expanded form with the outcome of the testSt the 
analysis of significant differences and the recommendations for system 
refinements J 
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Section 2 



SUMMARY 

Subsequent to the conduct of the reliability testing of 
instrumentation and procedures at five cooperating colleges, the 
COPES research study concentrated on three basic processes: 

• Determination of test results 

• Analysis of test results 

• Formulation of reconmendatlons, as indicated by the 
results, to strengthen the existing COPES design 

TEST RESULTS 

1. The Form 7 "summary profile" ratings of the two teams at 
each college visited were statistically correlated, and their cor- 
relation coefficients were found to be significant In all cases. 

2. The overall mean ratings of the two teams were virtually 
identical at three colleges (difference of .02 of a point or less on a 
five-point scale). At the college where the greatest difference 
occurred, the margin was less than half a point, 

3. The average per-ltem rating difference between the two tewis 
at each college ranged from a low of .37 of a point to a high of .73. 

4. The number of items rated equally by both teams at a college 
ranged from a high of 33 {out of the total of 60) to a low of 16. Of 
the grand total of 300 Individual -item rating comparisons at all five 
colleges, only 26 were more than one point apart on the five-point scale 
and only two more than two points apart. 

5. The number of Items which each team rated higher than the 
other at a college ranged from a "balance" of 16 and 16 (I.e., Team A 
rated 16 items higher than did Team B, and Team B rated 16 Items higher 
than did Team A) to an "imbalance" of 9 and 29. 



ANALYSIS OF TEST RESULTS 



1, In exploring possible reasons for rating "imbalance," it 
was learned that some teams had used different rating approaches than 
others. Some had adopted a "demanding" approach; others a "middle of 
the road" approach; still others at least a semi-'^kindly** approach. 
Some had based ratings solely on current performance; others had also 
taken into account improvements scheduled or In the planning stage. 
While these differences could not be demonstrated to be prime causes of 
"imbalance," they were identified as Important elements In the teams' 
major differences in rating Individual Items and, thus, as significant 
reliability factors, 

2, Other Important elements in major team rating differences 
were identified as inadequate heed or misunderstanding of criteria, 
failure to obtain sufficient inputs for well-considered judgments, 
misunderstanding of item intents and over-rel lance on the perceptions 
of a single team member, 

RECOMMENDATIONS INDICATED BY TEST RESULTS 

1. The approach to ratings should be standardized in terms of 
rigor (a middle area between the "demanding" and "kindly" approaches is 
suggested) and in terms of restriction to current performance or permit- 
ting consideration of planned improvements (the former is suggested for 
Form 7 ratings, with note to be taken of the latter in the teams' 
oral and written reports), 

2* Emphasis on full use of evaluation criteria should be 
continued* 

3, All Instrument evaluation Items and criteria should be 
reviewed to assure precision and understandablllty of wording* 

4* Site visit procedures should be revised as necessary to avert 
any over-reliance in team ratings on the perceptions of a single member, 
(Assignment of two members to share responsibility for making preliminary 
assessment applicable to each pc^ion of Form ^ Is suggested*) 

10 
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Section 3 



RELIABILITY TEST RESULTS 

CORRELATION COEFFICIENTS AND THEIR SIGNIFICANCE 

Coefficients of correlation were developed from the item 
ratings of the two teams at each of the five colleges.* These co- 
efficients, from low to high, were Cerntos College .44, Santa Monica 
College .49, West Valley College .66, Santa Ana College .68, and 
College of Alameda .76. These positive correlations between the ratings 
of the paired teams are considered statistically significant (i.e., not 
due to chance), since mathematically all exceed 99 chances In 100 that 
there is a positive correlation. The one percent level critical point 
{99 in 100) is .3308 and the five percent critical point (95 In 100) Is 
.2546. 

Since a perfect correlation between the ratings of two teams 
(probably never achievable) would be 1.0, the data suggest that further 
steps should be taken in system refinement and the education of team 
members to minimize human perceptual differences. , 

OTHER TEAM RATING COMPARISONS 

In addition to the statistical testing, the ratings Independently 
assigned by the two COPES teams to the 60 evaluation Items which make up 
the "summary profile" Instrument (Form 7) were compared for each of the 
five site visits Involved in the reliability study In five ways: 

• Overall mean (the difference between the teams' average 
rating for all 60 Items) 

• Overall mean point spread (the average per-item rating 
difference between the teams) 



♦Courtesy of James Stewart* research analyst. Division of Occupational 
Education, Chancellor's Office, California Community Colleges. 
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• Agreement-disagreement {the number of items rated 
differently, together with the extent of difference) 

• Hi^-low {the number of items rated higher by each 
team than the other team) 

• Individual-item point spread {the extent of difference 
on each item rated differently). 

Five-college averages or totals also were calculated. 

{AH ratings were on a five-point scale, ranging from 1.0, 
"poor," to 5.0, "excellent.") 

These additional comparisons were undertaken to study various 
aspects of alignment and non-alignment in the two teams' ratings at 
each site visit, so as to facilitate the isolation of discernible factors 
affecting reliability of system instrumentation and procedures (see 
Section 4, following). 

OVERALL DIFFERENCE IN MEAN RATINGS 

At three of the five colleges visited, the overall mean ratings 
of the two teams were virtually identical. At the college where the 
greatest difference occurred, the margin was less than half a point on 
the five-point scale. 

Table 1 summarizes the results- 
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Table 1 



DIFFERENCE IN OVERALL MEAN RATINGS BY TWO TEAMS 
AT FIVE COOPERATING COLLEGES 

ACTUAL 



INSTITUTION (o n five-point scale) PERCENTAGE 



Cerritos College 


.01 of a point 


0,25 


College of Alameda 


,11 of a point 


2,75 


Santa Ana College 


.02 of a point 


0,50 


Santa Monica College 


,02 of a point 


0.50 


West Vall^ College 


,43 of a point 


10,75 


Five-College Average 


.12 of a point 


2,95 



OVERALL MEAN POINT SPREAD 



The average per-item rating difference betvieen the two teams at 
each college ranged from a low of 0.37 point at College of Alameda to a 
high of 0,73 at West Vall^ College. 



Table 2 summarizes the complete results. 



Table 2 

AVERAGE PER-ITEM RATING DIFFERENCE 
BE7VEEN TWO TEAMS AT FIVE COOPERATING COLLEGES 

ACTUAL 



INSTITUTION (on five-Point scale) PERCENTAGE 



Cerritos College 


.53 of a point 


10,60 


College of Alameda 


.37 of a point 


7,40 


Santa Ana College 


.57 of a point 


11.40 


Santa Monica College 


.68 of a point 


13.60 


West Valley College 


.73 of a point 


14,60 


Five-College Average 


.58 of a point 


11.60 
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EXTENT OF AGREEMENT-DISAGREEMENT 



The number of items rated equally by both teams for each site 
visit ranged from a high of 33 (out of the total of 60) at College of 
Alameda to a low of 16 at Santa Monica College. Of the grand total of 
300 individual-Item rating comparisons at all five colleges, only 26 
were more than one point apart on the five-point scale. 

Table 3 sunvnarlzes the results: 

Table 3 

AMOUNT OF EQUAL AND DIFFERING RATINGS 
BY TWO TEAMS AT FIVE COOPERATING COLLEGES 
(N=60) 

DEGREE OF DIFFERENCE 
(by points on five-i>oint scale) 
INSTITUTION 0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4 



Cem' tos 


27 


12 


15 


3 


3 


0 


0 


0 


0 


Alameda 


33 


11 


15 


0 


1 


0 


D 


0 


0 


Santa Ana 


28 


4 


23 


2 


3 


0 


0 


0 


0 


Santa Monica 


16 


16 


22 


3 


2 


1 


0 


D 


0 


West Valley 


22 


2 


28 


3 


4 


0 


1 


0 


0 


Total (N^300) 


126 


45 


103 


11 


13 


1 


1 


0 


0 


Percent of Total 


42 


15 


34.3 


3.7 


4.3 


0.3 


0.3 


0 


0 



HIGH-LOW "BALANCE" 

Differing ratings at each site visit were studied to determine 
the number of items which each team had rated higher than the other team. 
The best "balance*' occurred at Santa Ana College, where, of 32 differing 
ratings, Team A was high on 16 and Team B high on 16, At West Vall^ 
College, on the other hand, of 38 differing ratings, TeMi A was high on 
only 9, while Te^ B was high on 29. 

Table 4 suirmarizes the complete results. 
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Table 4 



NUMBER OF HIGH RATINGS BY EACH TEAM 
ON ITEMS OF DIFFERENCE AT FIVE COOPERATING COLLEGES 



INSTITUTION 



TEAM "A" 



TEAM "B" 



Cerritos College 
College of Alameda 
Santa Ana College 
Santa hlonica College 
West Valley College 



17 
17 
16 
24 
9 



16 
10 
16 
20 
29 



ITEMS OF GREATEST POINT SPREAD 

The greatest rating difference {3 points) between two teams at 
TsiXT Visit" involved item 3.5 ("Balance between academic and occupational 
education interests on college curriculum committee"). 

Table 5 shows all items on which there was more than a one-point 

spread . 



la 



Table 5 

ITEMS OF GREATEST RATING DIFFERENCE 



MAXIMUM SPREAD SPREADS AT FOUR 
SINGLE COLLEGE OTHER COLLEGES TOTAL 
ITEM (Five-point scale) (Five-point scale) SPREAD 



Balance on curriculum conmlttee 


3.0 


1.5 


1.0 


1.0 


0 


6.5 


Provisions for handicapped 


2.5 


1.0 


1.0 


0 


0 


4.5 


Concurrence of programs, plan 


2.0 


1.5 


1.0 


1.0 


1.0 


6.5 


Provisions for disadvantaged 


2.0 


2.0 


1.0 


1.0 


0 


6.0 


Completions vs. enrollments 


2.0 


1.5 


1.0 


0.5 


0 


5.0 


Provisions in operating budget 


2.0 


1.0 


1.0 


1.0 


0 


5.0 


Administration's cormltment 


2.0 


1.0 


1.0 


0.5 


0 


4.5 


Planned enrollments vs. community 
needs 


2.0 


1.0 


1.0 


0 


0 


4.0 


Collection of needs information 


2.0 


1.0 


1.0 


0 


0 


4.0 


In-service education 


2.0 


1.0 


1.0 


0 


0 


4.0 


Participation In plan development 


2.0 


1.0 


0.5 


0 


0 


3.5 


Adequacy of facilities 


2.0 


1.0 


0.5 


0 


0 


3.5 


Individualized instruction 


2.0 


1.0 


0.5 


0 


0 


3.5 


Use of success/failure information 


2.0 


0 


0 


0 


0 


2.0 


Articulation with other Institutions 


1.5 


1.0 


1.0 


1.0 


0 


4,5 


Utilization of facilities 


1.5 


1.0 


1.0 


1.0 


1.0 


4.5 


Use of measurable learner objectives 


1.5 


1.0 


1.0 


0.5 


0 


4.0 


Job success of former students 


1.5 


1.0 


1.0 


0.5 


0 


4.D 


Qualifications of coordlnator(s) 


1.5 


1.0 


1.0 


0 


0 


3.5 


Provision for coordination 


1.5- 


1.0 


0.5 


0 


0 


3.0 


Salary schedule provisions 


1.5 


0.5 


0 


0 


0 


2.0 


Organization for coordination 


1.5 


0 


0 


0 


0 


1.5 
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Section 4 



DISCERNIBLE FACTORS AFFECTING RELIABILITY 

At all but one of the five colleges visited for purposes of 
the reliability study, the teams' overall perceptions of institutional 
performance in occupational education (as expressed in the difference 
of their average rating for all evaluation Items, shown In Table 1), 
were quite close. 

Yet the facts are that, at all but one college, average team 
ratings per Item disagreed by more than a half-point (Table 2) and that, 
again at all but one college, the teams disagreed on more than half of 
the 60 Items (Table 3). 

In view of those disagreements, how could the overall perceptions 
agree as nearly as they did? 

Of even greater importance, why did the disagreements occur and 
what might be done to lessen them In COPES' future? 

AMALYSIS 

The closeness of team's overall perceptions, despite their differ- 
ences on individual Items, can be largely explained by the "balance" of 
the differences (Table 4). 

Generally speaking, the sum-total effect of any team's ratings 
lower than those of the other team were pretty well offset by ratings 
higher than the other team. 

Note that the three site visits (Cerrltos College, Santa Ana 
College and Santa Monica College) with the least difference In overall 
mean ratings were also the three site visits with the best "balance" 
between the teams in number of high ratings on Items of difference. Note, 
too, that the site visit where the greatest difference In overall mean 
ratings occurred (West Valley College) was also the site visit of greatest 
"Imbalance," 

21 
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EXPLORATION OF REASONS FOR "IMBALANCE" 



Uniform team understandings of the Intent of each evaluation 
Item and of the criteria statements for Its rating probably can never 
be realized. 

Uniform team thoroughness of observation and data review relating 
to each Item probably can never be assured. 

Certainly these two factors are significant causes of rating 
differences (as Is shown under the next heading). But they can also 
be assumed, under normal circumstances, to have been relatively equally 
applicable to both teams at any of the site visits, and resultantly to 
have been part of the "balance" picture. 

Thus, reasons for "Imbalance" should be sought elsewhere. 

Two possibilities, based on past COPES experiences. Immediately 
suggested themselves: 

• Some teams may have made much less use of the criteria 
statements than others. 

• Some may have encountered unusual problems which 
prevented them from obtaining sufficient Inputs on a 
significant number of Items. 

One other possibility, never before generally considered, also 

emerged ; 

• Some teams may have taken a markedly different phlloso- 
phlcal appraoch to rating than others. 

To explore these possibilities, project staff queried the leaders 
of all 10 teams Involved In the site visits (see Appendix C). Four 
questions were asked: 
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• "Did your team base each rating of a Form 7 Item 
soundly on the criteria established for that Item?" 

• "Did your team generally adopt a 'demanding/ 'middle 
of the road* or 'kindly' approach to item ratings?** 

• '*D1d your team base its ratings solely on current 
institutional performance or also take Into account 
improvements scheduled to be made or In the planning 
stage?" 

• "Did your team have time to make enough observations and 
secure enough data for well-considered ratings on each 
item?" 

Team leaders' responses Indicated that all 10 teams had based 
all (or, in one case» "most") ratings on the criteria and that nine teams 
had had time to obtain sufficient Inputs for well-considered ratings. 

There were considerable differences^ however^ In the rating 
approaches taken by the teams. 

Three were "demanding" and seven "middle of the road/ with two 
of the seven on the "kindly" side of the middle and one more possibly on 
that same side. 

Five rated on^ or largely on^ the basis of **current performance/ 
the other five took into account "Improvements scheduled or In the planning 
stage." 

For the sake of the tidiness of analysis^ It would be nice to be 
able to demonstrate that these differences In approach were prime causes 
of "Imbalance*' between teams, but no such demonstration can legitimately 
be made. For example, at West Valley College^ the most likely site visit 
for establishing a causal relationship, Team A's "demanding" approach 
could probably be expected to produce lower ratings than Team B's "middle 
of the road» but did recognize extenuating circumstances" approach. On 
the other hand» Team A*s Inclusion of "scheduled and plannlng-stage 
improvements" could probably be expected to produce higher ratings than 
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Team B*s focus on "current performance,** Thus, there would be at least 
some offsetting effect, and any attempt to gauge the extent of offset 
would be sheer guesswork. 

Nevertheless, while the exploration may have arrived at a "dead 
end** in that regard, its identification of the different team approaches 
taken to the rating process made the undertaking decidedly worthwhile, 
since such differences can only detract from COPES* reliability {as is 
shown under the next heading), 

EXPLORATION OF REASONS FOR ITEM RATING DISAGPEEHENTS 

For all instances where ratings between the teams at a site visit 
differed by two or more points, team leaders and site visit chairmen were 
asked to provide explanations. 

Analysis of the explanations (see Appendix D) indicated the 
following reasons for disagreement: 

• Different team approaches (on the **demand1ng** to 
**kindly** scale) - 6 instances. 

• Inadequate heed or misunderstanding of criteria - 
5 instances, 

t Insufficient inputs - 3 instances, 

• Heavy reliance on planned improvements * 2 instances, 
t Misunderstanding of Item intent - one instance. 

In addition, one chairman cited the following as the reason for 
all significant differences between his two teams: 

t *'Every one, ,, can be traced to the perceptions of a particular 
individual on a given team who had been assigned the 
responsibility for n»k1ng a preliminary assessment applicable 
to a portion of the Form 7,,, I have concluded that one of the 
greatest weaknesses of the approach used is the fact that 
' only one member of a given team may have observed an event, 

and his or her judgment becomes the basis for the expression 
of consensus that everyone else Is obliged to support..,** 
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RECOHMENOflTIONS 



In the light of the foregoing discussion, the following actions 
are recoirmended to strengthen the existing COPES design: 

1. Standardize the approach to ratings^ In ternis of "demanding," 
"middle of the road" or '^kindly /' While there Is something to be said on 
behalf of both extremes (the rigorous and the supportive), a middle ground 
Is probably to be preferred. Seven of the 10 team leaders so Indicated; 
three favored "demanding," One team leader defined the middle ground as 
"conscientiously probing and non-Inspectorial." 

2. Standardize the approach to ratings. In terms of considering 
only "current Performance" or also taking Into account "Improvements 
scheduled to be made or In the planning stage .'* Team leaders were evenly 
divided) five to f1ve» on which approach they favored. Since It is often 
difficult to be sure when or whether a scheduled Improvement will actually 
be Instituted or a planned Improvement Implemented, perhaps the approach 
should be» as one team leader suggested, to rate on current performance^ 
but note significant planned Improvements In the oral and written reports. 

3. Continue emphasis on full use of evaluation criteria . 

4. Re-evaluate all Instrument Items and evaluation criteria to 
assure precision and understandablllty of wording . Particular attention 
might be given those Items on which there were the greatest aggregate 
rating differences at the five colleges Involved In the reliability stu^y 
{Table 5). 

5. Revise site visit procedures as necessary to avert artf over- 
reliance In team ratings on the perceptions of a single member . One 
means of achieving this would be to eliminate the current procedure of 
assigning each member to a special task (see page 3» "COPES Oetalled Site 
Visit Schedule," shown as Appendix B In the "Site Visit Hanual") and» 
Instead) have two members share responsibility for making preliminary 
assessment applicable to each portion of Form 7. 
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APPENDIXES 

Appendix A 

CALIFORNIA COMMUNITY COLLEGES COOPERATING IN THE STUDY 



College of Alameda 
Alameda 

Site Visit: January 22-24 



Cerritos College 
Norvtalk 

Site Visit: March 26-28 



Santa Ana College 
Santa Ana 

Site Visit: November 27-29 



Santa Monica College 
Santa Monica 

Site Visit: March 26-28 



West Valley College 
Saratoga 

Site Visit: February 20-22 
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Appendix B 



SITE VISIT TEAM MEMBERS PARTICIPATING IN THE STUDY 



College of Alameda 

John R. McKinley (Chairman), Dean of Administrative Services, Chabot 
College; John M, Hubbard (Team Leader), Assistant to Chancellor for 
Conmunity Relations, San Mateo Conmunity College District; Shirley B. 
McGillicuddy (Team Leader), Consultant; Rolf Bruckner, Associate Dean 
of Instruction, Gavilan College; Ross A. Carkeet, Jr., Instructor, 
Columbia Junior College; Dr. Wallace F. Cohen, Vice President-Instruc- 
tion, El Camino College; Chester P. Gromacki , Director of Vocational 
Education, North Orange County Conmunity College District; David D. 
Hurford, Director of Public Relations, Sears-Roebuck 4 Co.; Thomas D. 
Nesbitt, Consultant; Dr. John H. Rivers, Associate Dean-Student 
Personnel & Special Services, Monterey Peninsula College; Charles D. 
Rucker, instructor and Counselor, Southwestern College; J. Winston 
Silva, Specialist, Occupational Education, California Community 
Colleges; and Ted S. Sypolt, Specialist, Occupational Education, 
California Conmunity Colleges. 



Cerritos College 

Dr. Ray E. Loehr (Chaihnan); President, Ventura College; Dr. Ellsworth 
R. Briggs (Team Leader), Vice President for Instruction, College of the 
Redwoods; Dr. M. Jack Fujlmoto (Team Leader), Dean of Instruction, Los 
Angeles Pierce College; Edward Bratset, Educational Consultant; Fred E. 
Ittner, Associate Dean of Instruction-Occupational Education, Bakersfleld 
College; Shirley B. McGillicuddy, Consultant; David V. Robles, DepMy 
Director, Economic and Social Opportunities, Inc.; Julie Rosado, 
Specialist, Occupational Education, California Community Colleges; 
Harry Simonds, Educational Consultant; William B. Steinberg, Director 
of Vocational Education, San Diego Conmunity College District; James F. 
Stewart, Specialist, Occupational Education, California Community Colleges; 
Nino B. Valmassol, Instructor, Pasadena City College; and Burton T. Yount, 
Instructor, San Diego City College. 



Santa Ana College 

Dr. John C. Petersen (Chairman), President, Skyline College; Dr. Nathan 
H. Boortz (Team Leader), Director, Technical Education, Foothill Com- 
munity College District; C. Allen Paul (Team Leader), Dean of Technical- 
Vocational Education, Grossmont College; Charles C. Dahl, Associate Dean 
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Santa Ana College (Continued) 



of Instruction-Career Education, Ventura College; Louise S. Dyer» 
President, Board of Trustees, San Diego Conmunity College District; 
Dr. Robert E. Norton, President, Los Angeles Valley College; Thomas D, 
Nesbitt, Consultant; David V. Robles, Deputy Director, Economic and 
Social Opportunities, Inc.; Donald Y. Saguchi , Program Developer, East 
Los Angeles College; Herbert L. Schlackman, Assistant Dean, Occupational 
Education, Laney College; and Nino B. Valmassol , Instructor, Pasadena 
City College, 



Santa Monica College 

Jack Snyder (Chairman), Dean of Occupational and Continuing Education, 
Cabrlllo College; John M. Hubbard (Team Leader), Assistant to Chancellor 
for Community Relations, San Mateo Community College District; John V. 
Russo (Team Leader), Dean, Division of Science & Technology, Santa Ana 
College; Joseph E. Berruezo, Assistant Dean of Instruction, Vocational- 
Technical Education, College of Marin; Karen Bradstreet, A^lnlstratlve 
Associate, George Ebey Associates; Dr. Arthur N. Cherdack, Director, 
Educational Research and Analysis, Los Angeles Comnunlty College District; 
Louise S. Dyer, Lay Representative, Former Trustee, San Diego Comnunlty 
College District; Lenore H. Elsenstein, Chairman, Home Economics Depart- 
ment, Los Angeles Harbor College; Melvin J. Elklns, Assistant Dean, 
Vocational Education, Santa Barbara City College; George Hall, Specialist, 
CoirilKinity College Programs, California Coimiunlty Colleges; Jay B. La Foe, 
Assistant Vice President-Urban Affairs, Wells Fargo Bank; Edward J. 
Muraski, Director of Cooperative Education, Rio Hondo College; Manque 
Winters, Specialist, Community College Programs, California Community 
Colleges. 



West Valley College 

Shirley B, McGllllcuddy (Chairman), Shirley McGllllcuddy & Associates; 
C. Allen Paul (Team Leader), Dean of Technical-Vocational Education, 
Grossmont College; Herbert L. Schlackman (Team Leader), Assistant Dean, 
Occupational Education, Laney College; Edward Alfaro, Dean of Occupational 
Education, Santa Ana College; Roger D. Beam, Assistant Dean, Occupational 
Education, Compton College; Dr. Ellsworth R. Briggs, Vice President for 
Instruction, College of the Redwoods; Dale Collins, Programs Specialist, 
California Community Colleges; Dr. M. Jack Fujimoto, Dean of Instruction, 
Los Angeles Pierce College; Virginia A. Gries, Director, Home Economics, 
Pacific Gas & Electric Company; Don Richardson, Programs Specialist, 
California Community Colleges; Rollln M. Russell, Administrative Assist- 
ant to Vice President for Drilling & Production, McCulloch Oil Corporation 
Marjorle L. Sanchez, Director of Vocational Nursing, Sierra Community 
College; and John W, Tontsch, Assistant Professor-Coordinator, Los Angeles 
Pierce College. 
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Appendix C 

TEAM LEADER SURVEY - QUESTIONS AND RESPONSES 

1. DID YOUR TEAM BASE EACH RATING OF A FORM 7 ITEM SOUNDLY ON THE 
CRITERIA ESTABLISHED FOR THAT ITEM? 

Cerritos College 

Team A: "No. Most ratings were made by conmon-sense judgment. When 
disagreement arose within the team, we referred to the criterie for re- 
judging that Item." 

Team B: "Yes." 

College of Alameda 

Team A: "Yes." 

Team B: "Yes, If 'soundly' takes Into consideration Individual Inter- 
pretation of criteria which can't be absolute," 

Santa Ana College 

Team A: "Yes." 

TeamB: "Yes." 

Santa Monica College 

Team A: "Yes." 

TeamB: "Yes." 

West Valley College 

Team A-* "Yes." 

Team B: "Yes." 
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2d. DID YOUR TEAM GENERALLY ADOPT A "DEhWNDlNG»" "MIDDLE OF THE 
ROAD" OR "KINDLY" APPROACH TO ITEM RATINGS? 



Cerrltos College 
Team A: "Middle of the road." 
TeatnB: Viddle of the road." 
College of Alameda 

Team A: "Middle of the road, but leaning toward kindly." 
Team B: "Middle of the road» but not wishy-washy." 
Santa Ana College 
Team A: "Demanding." 

Team B: "Demanding; the te^ usually took a firm approach." 
Santa Monica College 

Team A: "On the kindly side of middle of the road." 
Team B: "Middle of the road." 
West Valley College 
Team A: "Demanding." 

Team B: "Middle of the road» but did recognize extenuating circumstances." 



Zb. (In response to a request to Indicate which approach they considered 
the most appropriate, seven team leaders Indicated they favored a 
middle ground, while three favored "demanding." One of those listed 
as a middle-ground advocate actually statedt "None of the above. 
Prefer to consider It conscientiously probing and non-Inspectorial." 
Other amplifying comments: "Demanding, but. In view of some recent 
observations, we should probably stress this with the college In 
advance of visit." "Middle of the road, but we should not hesitate 
to indicate areas of less-than-acceptable performance when so 
observed or perceived." 
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3a. DID YOUR TEAM BASE ITS RATINGS SOLELY ON CURRENT INSTITUTIONAL 
PERFORMANCE OR ALSO TAKE INTO ACCOUNT IMPROVEMENTS SCHEDULED TO 
BE MADE OR IN THE PLANNING STAGE? 

Cerritos College 

Team A: "Current performance." 

Team B: "Included scheduled and planning-stage Improvements." 
College of Alameda 

Team A: "Stuck pretty closely to current performance." 

Team B: "Included scheduled and planning-stage Improven^nts." 

Santa Ana College 

Team A: "Essentially current performance." 

Te^ B: "Included scheduled and planning-stage Improvements » particularly 
where a commitment had been made to rectify a problem or care for an unmet 
need. There were a few ^lingering' old conmltments In evidence but not 
Implemented which we discounted." 

Santa Monica College 

Team A: "Largely current performance." 

Team B: "Included scheduled and planning-stage Improvements." 
West Valley College 

Team A: "Included scheduled and planning-stage Improvements that had 
some solid commitment behind them." 

Team B: "Current performance." 



3b. (In response to ^ request to indicate which approach they considered 
the more appropriate, five team leaders favored "current performance" 
and five favored Inclusion of planned improvements. Amplifying comments 
were: "Ratings should accommodate the college's Identification of 
vulnerable or soft areas and their plans to change or correct such 
conditions." "Probably the latter, provided there Is clear evidence." 
"Rate on current performance; however, during the oral the reference 
to planned Improvements should be noted and briefly discussed/* 
"The latter, particularly where there Is a commitment to rectify a 
problem or care for an unmet need*") 
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4, DID YOUrv TEAM HAVE TIME TO MAKE ENOUGH OBSERVATIONS AND SECURE 
ENOUGH DATA FOR WELL-CONSIDERED RATINGS ON ALL ITEMS? 

Cerritos College 

Team A: "No, but I would say most all, so that judgments were not 
greatly in error," 

Team B: "Yes," 

College of Alameda 

Team A: "Yes." 

Team B: "Yes, with some qualifications." 

Santa Ana College 

Team A: "Yes, but barely." 

Team B: "Yes." 

Santa Monica College 

Team A: "Yes." 

Team B: "Yes," 
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Appendix D 

CAUSES OF MAJOR RATING DIFFERENCES BETWEEN SITE VISIT TEAMS - 
BASED ON EXPLANATIONS BY SITE VISIT CHAIRMEN AND TEAM LEADERS 

(All differentials of 2 or more points 
on the five-point scale considered) 

Differential Explanation 

3.0 Inadequate heed or misunderstanding of criteria 

by low-rating team. (Consensus: high rating.) 

2.5 Inadequate heed or misunderstanding of criteria 

and insufficient Inputs by low-rating team; 
heavy reliance on planned improvements by high- 
rating team. (Consensus: half-point above low 
rating.) 

2.0 Inadequate heed or misunderstanding of criteria 

by low-rating team; "kindly" approach by high- 
bating team. (Consensus: middle ground.) 

2.0 "^(Consensus: half-point below high rating.) 

2.0 "Demanding" approach by low-rating team; 

^ "middle of road" to "kindly" approach by high- 

rating team. (Consensus: middle ground.) 

2.0 Heavy reliance on planned Improvements by high- 

rating team. (Consensus: low rating.) 

2.0 Inadequate heed or misunderstanding of criteria 

and misunderstanding of Intent of Item by high- 
rating team. (Consensus: low rating.) 

2.0 "Demanding" approach by low-rating team. (Consensus 

half-point below high rating.) 

2.0 Inadequate heed or misunderstanding of criteria 

by high-rating team. (Consensus: half-point 
above low rating.) 

2.0 *{Consensus: half-point below high rating.) 

2,0 "Demanding" approach by low-rating team. "Middle 

of road" approach by high-rating team. (Consensus: 
middle ground.) 

2.0 Insufficient inputs by low-rating team. (Consensus 

half-point below high rating.) 
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Differential 



Explanation 
^(Consensus: middle ground.) 



2.0 



2.0 



"Kindly" approach and/or insufficient inputs 
by high-rating team. (Consensus: low rating.) 



2.0 



"Oemanding" approach by low-rating team. 
(Consensus: higti rating.) 



*Site visit chairman's explanation : "Every one of these differences can 
be traced to the perceptions of a particular individual on a given team 
who had been assigned the responsibility for making a preliminary assess- 
ment applicable to a portion of the Form 7. As you know, when each team 
goes over the Form 7, the final rating decided upon for each item is 
usually the result of an expression of consensus, with a particular evalu- 
ator's observation or impression serving as the core." 

"I have concluded that one of the greatest weaknesses of the approach 
used is that which stems from the fact that only one member of a, given 
team may have observed an event, and his or her judgment becomes the basis 
for the expression of consensus that everyone e1$e is obliged to support. " 

"Some circumstances on a college campus are so readily apparent to every 
member of the team that it is easy for all team members to support a con- 
sensus statement. In other instances, the judgment of the team is sub- 
stituted for that of the team member who observes a particular event and 
reports on it. The reason why each team scored the items as they did is 
less to be found ln the judgments of Uie total team than in the reported 
observations of a team member whose observations become Uiose of tne team . 
Most of the differences noted here, in my judgment, are grounded less in 
fact than in the feelings certain team members had. about items they were 
supposed to specifically observe and report on to the rest of their 
colleagues." 
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